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Students typically view their enrollment in college as an oppor- 
tunity to acquire the knowledge and skills that are necessary to 
succeed in the workplace and to advance their general knowl- 
edge and life skills (Carter, Bishop 8c Kravits, 2006). The rate 
of entry into college provides “an indication of the degree to 
which a country’s population is acquiring higher-level skills and 
knowledge” (Wirt et al., 2004, p. 62). However, in 2004, one in 
four college freshmen at 4-year universities did not return for 
their sophomore year and nearly half of students in commu- 
nity colleges did not return to complete their degree (Netscape 
News with CNN, 2004). Empirical research over the last several 
decades corroborates these findings (Astin, 1975; Tinto, 1993). 
Tinto described this as a tremendous loss of resources (i.e., talent 
and revenue) and a principal concern for students, parents, and 
administrators. As a result, many universities and colleges have 
implemented some form of intervention, formal or informal, to 
increase academic achievement and positive social adjustment. 
These efforts are focused on increasing retention and graduation 
rates. 
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First Year Experience (FYE) programs have been implemented in uni- 
versities and colleges to address the emergent needs of matriculating 
students. It is well documented that many students are not prepared for 
the rigors of postsecondary education and FYE programs are designed 
to supplement the necessary academic and life skills. These academic 
and life skills range from knowing the location of the library to seek- 
ing out appropriate personnel when personal issues arise. Evaluating 
programs developed to address such a wide range of knowledge and 
skills is an obvious challenge, especially when each FYE is customized 
for each corresponding university or college. 

In the case of one FYE program at a large university, evaluation of 
student Knowledge, Attitudes, and Behaviors (KAB) provided a concep- 
tual model to examine the program as a whole. This particular study 
revealed the following about the FYE program: 

• The KAB approach is pragmatic and efficient. 

• FYE programs are capable of increasing knowledge, which is a 
principle objective of the courses. 

• The FYE program impacted attitudes associated with academic 
and life skills. 

• Males and females participate and respond to the FYE differently. 
For example, females reported significantly higher attitudes than 
males. 

• The study of FYE Programs overall is highly complex. Further longi- 
tudinal studies focusing on retention and attrition are needed. 


Schrader, P. G., & Brown, S. W. (2008). Evaluating the first year experience: Students' 
knowledge, attitudes, and behaviors . Journal of Advanced Academics, 19, 310-343. 
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EVALUATING A FIRST YEAR EXPERIENCE PROGRAM 


This study examines an intervention program that has been 
developed for first-year college/ university students to facilitate the 
transition to postsecondary education. The First Year Experience 
(FYE) program has been in place at a large Northeastern uni- 
versity for more than 10 years and is designed to facilitate the 
acquisition of traditional academic and social skills. These skills 
vary widely but include study skills (e.g., note taking, library 
research), time management skills, institutional awareness (e.g., 
location of the library), and appropriate interpersonal behavior 
(Kuo, Hagie, 6c Miller, 2004; Kuh, 1995; Terenzini, Pascarella, 6c 
Blimling, 1996; Zhao 6c Kuh, 2004). The FYE program addresses 
these topics via a one-credit elective course that incorporates 
personal speakers, seminars, and online collaboration. Given the 
complexity and variety of topics associated with the program as 
well as the complexity, a multiple construct approach was used 
to evaluate the FYE program. Specifically, this study examined 
students’ change in knowledge, attitudes, and behaviors (KABs) 
associated with these issues over time (see Schrader 6c Lawless, 
2004, for more information on the KAB approach to program 
evaluation). 


Review of the Literature 

Factors of a Successful Collegiate Experience 

By understanding the entry characteristics of newly matricu- 
lated college students and their developmental experiences, we 
are better equipped to facilitate student development and growth. 
Terenzini et al. (1994) identified several variables that influence 
this transition, including academic and social involvement, fam- 
ily background, peer group, socioeconomic status, and academic 
preparation. According to researchers, involvement is the extent 
to which a student participates in academic as well as nonaca- 
demic (e.g., social) activities (Kuh, 1995; Terenzini et al., 1994). 
Astin (1984) identified academic involvement, involvement with 
faculty, and involvement with peer groups as three of the most 
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influential forms. Kuh found that students attribute involvement 
in certain activities (i.e., leadership, peer, academic, faculty, work, 
travel, or ethos related activities) as positive developmental fac- 
tors in college. Further, according to Kuh, “the curriculum is, 
without dispute, the organizing framework for academic insti- 
tutions. At the same time, students benefit in many ways from 
out-of-class experiences, ranging from gains in critical thinking 
to relational and organizational skills . . .” (pp. 149-150). 

Whether from in-class or out-of-class experience, researchers 
acknowledge that academic and social skills, along with learning 
and thinking strategies, play a significant role in academic suc- 
cess. Dickinson and O’Connell (1990) advocated high-quality 
time on task, while White and Shahan (1989) focused on the 
broader topic of motivation. Ting, Grant, and Plenert (2000), 
Turner (1992), and Zhao and Kuh (2004) stressed the impor- 
tance of involvement in social activities and involvement in 
academic counseling as factors in collegiate success. In a more 
recent study, Kuo et al. (2004) examined several note-taking, 
study, test-taking, technology, and social skills in relation to stu- 
dent success and the manner in which students behave in order 
to achieve that success. Kuo et al. found that in academic areas, 
students often work individually, relying on their existing skill 
sets rather than utilizing campus resources or social groups to 
achieve success. With respect to technology, Miller and Pope 
(2003) reported that universities have increased their expecta- 
tions and often require students to have their own personal com- 
puter. Collectively, these studies reflect the range of academic 
and social skills that are vital for student success in college. 

Unfortunately, research indicates that some students cur- 
rently enrolled in colleges and universities are not prepared to 
meet the academic and social challenges of postsecondary edu- 
cation. For example, Wirt et al. (2004) reported that in the fall of 
2000, approximately 76% of postsecondary institutions offered 
some form of remediation in the area of basic skills (i.e., reading, 
writing, or mathematics), suggesting that many students enter- 
ing postsecondary education are underprepared for the academic 
demands of college. Pitts, White, and Harrison (1999) found 
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that many professors perceive a decrease in student motivation 
to learn and a willingness to adopt a passive learning approach, 
as compared to a generation ago. Even if they are prepared aca- 
demically, students may neither be aware of, nor use, existing 
services (e.g., technology labs, libraries, counseling services) to 
their advantage (Crismore, 1984). These issues are compounded 
by the fact that at a postsecondary level students suddenly depart 
a familiar teacher-directed environment and enter a student- 
directed environment (Wratcher, 1991). In college, students 
must manage their time and allocate their own resources toward 
completing their assignments, which, as Wratcher observed, is 
often difficult for freshmen. If these factors are not addressed 
adequately, they present an obstacle for everyone involved. 
Students who do not successfully manage their first year “stop- 
out” or drop out of school entirely (Astin, 1975; Tinto, 1993). 

Fortunately, there are many learnable skills and traits that 
influence success in college. Garavalia and Gredler (2002) 
reported that several factors are linked to student performance 
including prior GPA, SAT scores, and organization and plan- 
ning. However, unlike SAT scores and prior GPA, “organization 
and planning can be manipulated” (p. 624). In circumstances 
where factors lie outside the direct influence of a university (e.g., 
family background, peer group, and socioeconomic status), addi- 
tional local, college/university, and federal resources help provide 
support for the students (Sonnenberg, 2004). Whether univer- 
sities elect to add resources or provide direct instruction with 
respect to trainable skills, solutions exist to facilitate students’ 
adjustment to their first year in postsecondary education. 

First Year Experience Programs 

In response to the growing need to prepare students for their 
first year, many universities have developed programs and initia- 
tives intended to facilitate the transition to college. Although 
early efforts took the form of increased institutional resources 
such as libraries, writing centers, computer labs, or personal 
computers (Miller & Pope, 2003; Pascarella SeTerenzini, 1991), 
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universities and colleges also have offered courses and seminars 
focused on the development of academic skills and social skills 
(Howard & Jones, 2000). Whether the courses were offered as 
formal requirements or an option, they are almost exclusively 
designed to impart information (i.e., knowledge) and experience 
to students (Barefoot 6c Gardner, 1993; Gelb, 2007; Howard 6c 
Jones, 2000; VanderStoep 6c Pintrich; 2008; Wratcher, 1991). 
These curricular initiatives have taken the name: First Year 
Experience programs (or FYE). 

White, Goetz, Hunter, and Barefoot (1995) were among the 
first researchers to describe the FYE intervention strategy in 
detail for incoming first-year students. Their research focused not 
only on the assessment of first-year students’ skills and degrees 
of success, but also on the design and implementation of the 
program. White et al. suggested that the FYE should afford stu- 
dents opportunities to interact socially (with peers and faculty) 
as well as introduce students to academic facilities, counseling 
staff, and other faculty during advising/planning sessions. They 
described program content, structure, and goals. The FYE pro- 
gram “should be understood not as single events, but as processes 
that should be linked programmatically” (White et al., 1995, p. 
33). This early work has informed FYE programs nationwide, 
including the FYE program currently under investigation. 

In addition to describing the various program objectives and 
curricula (e.g., study skills, social skills), White et al. (1995) indi- 
cated that FYE programs must be tailored to suit the needs of 
the university that sponsors it. Unfortunately, this presents obvi- 
ous challenges in terms of program evaluation. Universities vary 
in many ways, as do the corresponding FYE programs and course 
topics, and evaluation strategies must be developed to suit those 
particular needs. At the same time, assessment strategies must 
remain comprehensive and address the FYE program as a whole. 
Because of the complexities involved, researchers have argued 
that a multiple construct approach is appropriate when evaluating 
interventions (Schrader 6c Lawless, 2004). One multiple construct 
approach in particular, the Knowledge, Attitudes, and Behavior 
(KAB) methodology, has been argued to be efficient and effec- 
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tive when evaluating complex, multifaceted interventions like the 
FYE program (Schrader & Lawless, 2004). This approach evalu- 
ates participants’ cognitive, affective, and behavioral constructs as 
they pertain to an intervention. Although other variables may be 
of interest, these constructs are of interest because they have been 
shown to be interrelated (Alexander, 1992; Alexander, Jetton, & 
Kulikowich, 1995) and important factors in the learning process 
(Alexander, 2003). Because of its ability to measure multiple con- 
structs holistically, the KAB methodology is an appropriate evalu- 
ation strategy for the FYE program. 

Gender 

Although White et al. (1995) recommended that FYE pro- 
grams should be evaluated holistically, universities often focus on 
the first-year population as a whole. They adopt a “one-size-fits - 
all mentality about serving college students” (Kuo et al., 2004, p. 
65) that does not reflect the variability across institutions. More 
importantly, this evaluation approach neglects the ways in which 
males and females have been shown to differ on many cognitive 
tasks (Caplan 6c Caplan, 1997; Hutt, 1972; Hyde 6c McKinley, 
1997; Kimura, 1999; Richardson, 1997). For example, there is a 
tendency for girls to outperform boys across grades and across 
some subject areas as they mature (Dwyer Sc Johnson, 1997; 
Pomerantz, Altermatt, Sc Saxon, 2002). These areas include 
mathematics, English, and technology (American Association 
of University Women, 1996; Hyde, Fennema, Sc Lamon, 1990). 
Although these differences in mathematical and verbal ability 
continue to persist, the magnitude of these differences appears 
to be declining (Halpern, 2000; Reis, 1998). However, this per- 
formance appears to come at some cost. It has been suggested 
that while younger females, on average, achieve higher academic 
grades than males, they also are susceptible to increased internal 
distress and anxiety related to academic preparation and perfor- 
mance (Pomerantz et al., 2002). 

At the university level, males and females have been shown to 
exhibit differences in the ways they cope with stress (Lawrence, 
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Ashford, 6c Dent, 2006). Males on average report higher self- 
esteem as a result of their coping style (emotional inhibition), 
while females report higher levels of attainment. Beyond cog- 
nitive and psychological factors, gender continues to emerge as 
an important issue. Specifically, the demographics of universi- 
ties and colleges are changing. Women were awarded 57% of all 
bachelor’s degrees in the U.S., a statistic that has steadily risen 
since 1970 (Wirt et al., 2004). Given the quantity of research in 
the area and the changing student population, developing and 
evaluating FYE programs also must examine the possible differ- 
ences between males and females. 

Collectively, all students face adjustment issues, and steps 
should be taken to develop appropriate interventions (Garner 6c 
Jewler, 1992; Wratcher, 1991; Zhao 6c Kuh, 2004). These inter- 
ventions are directed toward the skills and knowledge that will 
enable students to adjust to college and to be successful both 
academically and personally. However, only through under- 
standing student needs will we be capable of designing effective 
interventions. For these reasons, we were interested in imple- 
menting a KAB battery in an effort to evaluate the effectiveness 
of the FYE program. Specifically, the following sets of research 
questions were addressed: 

1. Are there significant differences in the ways students 
enrolled in the FYE program and a comparison group 
respond to the KAB battery? Is there a difference over 
time? 

2. Are there significant differences in the way males and 
females respond to the KAB battery? Is there a differ- 
ence over time? 


Methods and Procedures 

Current Contexts 

This research took place in the context of the FYE at a large 
Northeastern university in the United States. The FYE pro- 
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gram begins with a 3 -day student orientation session introduc- 
ing matriculating students to the physical layout of the campus, 
socializing them with a group of their peers, and introducing 
them to the academic environment. During orientation, students 
meet with advisors, schedule courses, and take placement tests. 
They also participate in various social and community building 
activities. Following orientation, the majority of matriculating 
students (approximately 65% in 2002 rising to 88% in 2006) 
enroll in an FYE course during their freshman year. 

The FYE program is offered as a one-credit course with mul- 
tiple sections. Each section addresses common program-wide 
instructional objectives and topical issues related to specific sub- 
groups within the university. For example, engineering students 
are encouraged to enroll in an engineering seminar where they 
will learn about specific course requirements, while honors stu- 
dents enroll in sections developed for them. University faculty 
and staff knowledgeable in their particular content area and in 
student advising, teach the various sections. In some cases, sec- 
tions are cotaught with a student facilitator who recently com- 
pleted a leadership course. Although instruction in each section 
is variable, every section addresses the same overall content and 
instructional objectives (e.g., academic skills, library use, and 
social conduct), thereby maintaining programmatic consistency. 
A characteristic of the FYE program at this institution is small 
class size, 18 students or fewer. 

Utilizing technological and face-to-face resources, the FYE 
program is designed to enhance a student’s time management, 
communication, social interaction, and study skills, as well as to 
help develop strategies for problem solving and critical thinking. 
Students get to know a faculty member or a professional staff 
person to whom they can turn for advice and support in the 
future. Implemented in this way, the FYE is structured to impart 
a set of academic skills as well as to foster a sense of community. 
Both of these global objectives have been shown to be vital in the 
transition into postsecondary education. 
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Table 1 

Frequency of Gender by Treatment Group 


Student Group 

Gender 

Female 

Male 

Total 

FYE 

461 

209 

670 

Comparison 

141 

93 

234 

Total 

602 

302 

904 


Participants 

Approximately 65% of incoming freshmen voluntarily 
enrolled in the First Year Experience (FYE) program at a large 
Northeastern university in the fall of 2002. The remaining 35% 
of incoming freshmen opted not to participate in the FYE pro- 
gram and therefore served as the comparison group. Data were 
collected from 2,768 entering freshmen (which accounts for 
approximately 87% of the 3,185 person entering freshman class) 
during their orientation sessions and from 1,141 students at the 
end of their first semester. Although all students enrolled in the 
FYE program took part in the curriculum activities, only those 
who provided informed consent were included in this study. 

Initial screening of the data revealed a total of 904 usable, 
matched (pre and post) data points. A total of 670 (74.1%) stu- 
dents enrolled in the FYE while the remaining 234 (25.9%) 
students served as a comparison group. The total sample was 
comprised of 602 females (66.6%) and 302 males (33.4%). The 
frequency of gender by student group is provided in Table 1. 
Overall enrollment characteristics for the Fall of 2002 are pre- 
sented in presented in Table 2. 

Instrumentation 

A battery of self-report scales measured the participants 
along three theoretical dimensions, Knowledge, Attitudes, and 
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Table 2 

Overall Student Characteristics for Fall 2002 


Student Characteristics 

Undergraduate — 1 8,662 
(3,185 freshmen) 

Grad/ 

Professional — 7,1 80 

Female 

53% 

53% 

Male 

47% 

47% 

Minority 

17% 

12% 

International* 

1% 

18% 

In-State Residents** 

80% 

70% 

Full-Time Degree 

85% 

57% 

Part-Time Degree 

9% 

33% 

Non-Degree (FT &PT) 

6% 

10% 


*107 countries were represented in the 2001—2002 international student population. 
**76% of undergraduates on main campus are in-state residents. 


Behaviors (KABs; Schrader 8c Lawless, 2004). The three differ- 
ent scales were administered together. Each scale addressed a 
different theoretical dimension (i.e., Knowledge, Attitudes, and 
Behaviors) as they pertained to academic and life skills of college 
freshmen. The Knowledge and Attitude scales each contained 22 
Likert-type items using a 5 -point Likert response format, rang- 
ing from (1) strongly disagree to (5) strongly agree. The Behavior 
scale contained 18 items at pretest and also was formatted as a 
Likert (1) never to (5) frequently with a not applicable selection 
(no value). A list of the items for each scale from the battery is 
presented in Appendix A. 

Instrument Administration 

The pretest battery was administered to all students attend- 
ing the LYE orientation in paper-and-pencil format. The post- 
test battery was administered to students via the Web at the 
conclusion of the fall semester. LYE program instructors used 
WebCT® during the course to facilitate data collection (e-mails, 
instructor request, etc.). Small incentives were offered for the 
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completion of the instrument battery (i.e., gift certificates to the 
college bookstore). Students in the comparison group had no 
access to the FYE WebCT® site and were therefore contacted 
via e-mail and through professors in other general freshman year 
courses (e.g., introductory psychology). 

Preliminary Data Analysis 

Previous implementations and analysis of this battery have 
revealed that the scores on each scale (i.e., Knowledge, Attitudes, 
and Behaviors) show evidence of reliability and validity for use 
with a college population (Schrader, 2003; Schrader 6c Brown, 
2004). These implementations also revealed that each scale was 
multidimensional. According to these earlier investigations, the 
scores from the Knowledge scale were grouped into four distinct 
and stable components, the scores from the Attitude scale were 
grouped into two components, and the scores from the Behavior 
scale were grouped into four components. Table 3 presents these 
components (i.e., variables), their reliabilities, and the items that 
corresponded to them. 

According to Stevens (1996), principal component analysis 
is a psychometrically sound procedure to reduce a large number 
of predictors (15 to 40) without detracting from one’s ability 
to draw inferences from the results. Following this advice, raw 
scores on the original scaling were used in the analysis (Grice, 
2001). In this case, summing the responses for the scale and then 
dividing the total score by the number of items for each com- 
ponent created the scale scores. This process created scale scores 
that remained on the original 5 -point scale, enabling compari- 
sons across conditions and scales. Using scale scores allowed 
greater parsimony within the multivariate model. The means and 
standard deviations with respect to student groups are presented 
in Table 4. The means with respect to gender are presented in 
Table 5. 

Initial screening of the data did not reveal any statistical 
problems with the data. However, using the metrics described by 
Tabachnick and Fidell (1996), some of the variables posed a pos- 
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Table 3 


Component Variables and Corresponding Items 


Knowledge Components and Description 

Items 

Alpha 

Knowledge of Resources: Items pertained to the use of 
resources, including technological resources. 

6,11,12, 

13,17,18 

.76 

Knowledge of Academic Skills: Items that pertained to 
academic issues (e.g., note-taking, academic goal setting, 
and school effort). 

1,2, 3, 9, 
16 

.79 

Knowledge of Health and Wellness: Items referred to issues 
of personal and social awareness, general health issues, or 
medical issues. 

14, 15, 19, 
20,21,22 

.82 

Decision-Making Knowledge: Items dealt with the 
decision-making process, from understanding influences 
involved in decisions (e.g., ethical factors or personal 
limitations) to the outcomes of decisions (e.g., setbacks or 
conflicts). 

4, 5, 7, 8, 

10 

.74 

Attitude Components and Description 

Items 

Alpha 

Attitude Toward Tools for Collegiate Success: Items related to 
general tools for academic or collegiate success including 
issues of diet and exercise. 

1,2, 3, 9, 
16,17,18, 
19,20,21 

.93 

Attitude Toward Interactions: Items pertained to 
interactions one might experience on campus and the 
influences critical in those interactions. 

4, 5, 6, 7, 
8, 10, 11, 
12, 13, 14, 
15,22 

.92 

Behavior Components and Description 

Items 

Alpha 

Academic Behaviors: Items pertained to academic issues 
(e.g., note-taking, academic goal setting, and school 
effort). 

1,2, 3, 8, 
13 

.80 

Decision Behaviors: Items related to the decision-making 
process, from understanding influences involved in 
decisions (e.g., ethical factors) to the outcomes of 
decisions (e.g., setbacks or conflicts). 

4, 5, 6, 7 

.74 

Conscientious Behavior: Items pertained to issues of 
appropriate behavior in differing circumstances (e.g., 
health-related, social, or academic) . 

9, 10, 11, 
12, 14 

.61 

Proactive Behavior: Items reflected initiatives that 
students undertake in order to facilitate health related and 
interpersonal well-being. 

15,16,17, 

18 

.71 


Note. Cronbach’s Alpha used as the estimate of reliability. 
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Table 5 
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sible violation to normality. Specifically, Knowledge of Health 
and Wellness and Attitude Toward Tools for Collegiate Success 
were found to have skewness and kurtosis statistics in excess of 
twice their respective standard errors. Although Stevens (1996) 
suggested several options for transforming variables to achieve 
normality, interpretation of these variables is difficult, at best. 
Although normality usually enhances a multivariate solution, it 
is not always required for analysis (Tabachnick 8c Fidell, 1996). 
In this case, all of the variables followed similar trends. These 
trends may be explained by the fact that most participants’ self- 
report scores on the Likert-type scales are high and scores are 
somewhat homogenous (i.e., negatively skewed and leptokurtic). 
Regardless, all inferences should be interpreted with caution. 

Design 

This study followed a quasi-experimental, nonequivalent 
comparison group design. To address the research questions, 
a doubly multivariate, repeated measures analysis of variance 
was applied to the data. This approach was selected because: 
(a) Participants’ responses were measured at two times (prior to 
instruction and after instruction at the conclusion of the semes- 
ter); (b) multiple measures were administered on each occasion; 
and (c) there are two sets of groups involved in this analysis (i.e., 
student group and gender). For this analysis, time served as the 
within-subjects factor while both student group and gender 
served as between-subject independent variables. The 10 derived 
component scale scores served as the dependent variables in the 
analysis (see Appendix B for the intercorrelation matrix). 


Results 

Research Question One: Student Group 

Between-subjects analysis revealed a significant main effect 
for student group, Wilks’ A = .973, A (10, 892) = 2.465, /> = .007, 
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partial r\ 2 = .027. Follow-up tests revealed differences on one 
variable, Attitudes Toward Interactions, F( 1, 901) = 4.433 , p = 
.036, partial r\ 2 = .005. However, while the main effect and 
follow-up test were statistically significant, the estimate of the 
effect size was low (Cohen, 1988). This suggests that although 
there is statistical significance, there is limited practical signifi- 
cance associated with these findings. 

To further explore the relationship between the FYE and 
comparison group over time, the interaction was examined. 
Analysis revealed a significant interaction over time with respect 
to student group, Wilks’ A = .970, A (10, 892) = 2.733 ,p = .003, 
partial r\ 2 = .030. Follow-up tests revealed differences on two 
variables, Knowledge of Resources, A(l, 901) = 11.979, p = .001, 
partial r\ 2 = .013, and Attitudes Toward Interactions, A(l, 901) = 
4.773, p = .029, partial r\ 2 = .005. For Knowledge of Resources, 
follow-up analyses of the simple effects were conducted using 
the estimated marginal means and revealed a significant change 
in scores over time for both the FYE and comparison groups 
(see Table 6). However, although there was no significant dif- 
ference between groups’ responses during the pretest, there was 
a significant difference between groups’ responses in favor of the 
FYE group during the posttest (see Table 7). While both groups’ 
scores increased overall, there was a larger positive change with 
respect to the FYE group. Figure 1 portrays the profile plot for 
this interaction. 

For Attitude Toward Interactions, follow-up analyses of 
simple effects revealed a change in scores over time for the com- 
parison group only (see Table 6). Further, although there was 
no significant difference between groups at pretest, there was a 
significant difference at posttest (see Table 7). Evaluation of esti- 
mated marginal means and the profile plots (see Figure 2), indi- 
cated that the comparison group’s scores on Attitude Toward 
Interactions decreased significantly while the FYE group’s scores 
remained relatively unchanged. However, the estimate of effect 
size for this follow-up test was also low. This implies caution 
when drawing practical inferences from this statistic. 
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Table 6 


Student Group Pairwise Comparisons Over Time 



Student FYE 

Mean Difference 

Standard 


Variable 

Grouping 

(Pretest-Posttest) 

Error 

Significance 

Knowledge 
of Resources 

FYE 

-.328 

.033 

< .001 


Comparison 

-.116 

.053 

.030 

Attitude 

Toward 

FYE 

.043 

.027 

.114 

Interactions 

Comparison 

.154 

.044 

< .001 


Note. Bonferroni adjustment used for multiple comparisons. Analysis based on Estimated 
Marginal Means. 


Table 7 


Pairwise Group Comparisons on Different Occasions 


Variable 

Time 

Mean Difference 
(FYE-Comparison) 

Standard 

Error 

Significance 

Knowledge of 

Pretest 

-.061 

.053 

.249 

Resources 

Posttest 

.152 

.049 

.002 

Attitude Toward 

Pretest 

.021 

.046 

.648 

Interactions 

Posttest 

.132 

.042 

.002 


Note. Bonferroni adjustment used for multiple comparisons. Analysis based on Estimated 
Marginal Means. 


Research Question Two: Gender 

Between-subjects analysis revealed a significant main effect 
with respect to gender, Wilks’ A = .880, ff(10, 892) = 12.167,/>< 
.001, partial r\ 2 = .12. Test ofbetween-subject effects with student 
group averaged revealed differences on each dependent variable 
with the exception of Proactive Behavior. Table 8 presents the 
univariate tests for this main effect. Although there were many 
differences between the ways males and females responded to 
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• FYE 

- Comparis 


Pretest Posttest 

Time 


Figure 1. Estimated marginal means for Knowledge of Resources. 
Students enrolled in the FYE course reported significantly higher 
scores at posttest. 


■FYE 

— Comparis 


Pretest Posttest 

Time 


Figure 2. Estimated marginal means for Attitudes Toward 
Interactions. Students in the comparison group exhibit a signifi- 
cant decline in their ratings. 


328 Journal of Advanced Academics 


Schrader & Brown 


Table 8 

Univariate Tests of Gender (n = 904) 


Variable 

Mean 

Square 

F 

(1,901) 

Sig. 

Partial 

r\ 2 

Knowledge of Resources 

2.721 

9.441 

.002 

.010 

Knowledge of Academic Skills 

6.357 

20.390 

< .001 

.022 

Knowledge of Health and Wellness 

1.683 

6.752 

.010 

.007 

Decision Making Knowledge 

1.084 

4.282 

.039 

.005 

Attitudes Toward Tools for Collegiate 
Success 

9.470 

44.884 

< .001 

.047 

Attitude Toward Interactions 

9.223 

40.334 

< .001 

.043 

Academic Behaviors 

21.236 

71.619 

< .001 

.074 

Decision Behaviors 

2.388 

8.525 

.004 

.009 

Conscientious Behavior 

2.620 

13.607 

< .001 

.015 

Proactive Behavior 

.681 

1.951 

.163 

.002 


Note. This test is based on the linearly independent pairwise comparisons among the esti- 
mated marginal means. 


the KAB battery, the interaction between time and gender was 
not significant. This suggests that although responses differ on 
average, those differences are stable over time. 


Discussion 

The first year of college presents a variety of challenges for 
students and faculty. The literature indicates that a wide range of 
issues, from social to individual factors, influence students’ suc- 
cess during their first year. It follows that FYE programs are 
challenged to provide experience and instruction across this 
chasm of ideas (White et al., 1995). Because the variability of 
topics and student needs is so diverse, evaluating the effect of 
FYE programs is highly complex and involves a large number of 
variables. As a result, the current study focused on a small subset 
of the possible variables that contribute to student success. The 
results of this investigation do not necessarily support the value 
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of the FYE program, especially given the nonrandom nature of 
the assignment variable. 

When interpreting the results of this investigation, one must 
consider the importance of effect size in relation to practical and 
statistical significance (Cohen, 1988). The data from this investi- 
gation do not support the conclusion that the FYE is valuable in 
terms of its programmatic objectives. Although there are many 
statistically significant differences between groups, the effect sizes 
of these differences are small. Therefore, one cannot conclude 
that the FYE program is effective. However, Cohen used the 
concept of practical significance to promote judicious and mean- 
ingful interpretation of research results. As such, we acknowl- 
edged the practical limitation associated with these findings but 
suggest that there is a link between being in the FYE program 
and learning. Unfortunately, the self-selected nature of participa- 
tion in the FYE program also potentially confounds the results 
of the study. As a result, we are unable to suggest programmatic 
change or courses of action based on these data. Nevertheless, we 
assert that the trends are found in areas of considerable interest 
and provide grounds for future investigations. 

Statistically, there are significant results associated with the 
way students report their Knowledge of Resources; students in 
the FYE group demonstrated greater gains over time. Further, 
students engaged in an academic experience without FYE 
instruction did not exhibit such gains. Similarly, researchers have 
asserted the importance of resource awareness (Crismore, 1984; 
Terenzini 6c Reason, 2006) and argued that student acclima- 
tion to the facilities, support services, and other resources is a 
principal goal of the FYE program (Barefoot 6c Gardner, 1993; 
Howard 6c Jones, 2000). Because differences exist in an area of 
considerable interest, trends apparent in these data should be 
examined in greater detail in future investigations. 

While less straightforward, results associated with Attitudes 
Toward Interactions revealed other notable differences. With 
respect to this variable, the FYE group remained relatively stable 
with only a slight decline in their ratings while the scores for the 
comparison declined significantly. A principal goal of FYE pro- 
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grams is to establish a social cohort early in students’ academic 
careers and provide information about interactions with other 
individuals as well as various university systems (e.g., financial aid, 
counseling, health). Without this experience, students appear to 
lower their appraisal of these important resources. Unfortunately, 
the reason for this decline in the general population is not known. 
Similarly, it is unclear whether the FYE program is effective in 
staving off any significant decline in these attitudes. 

Given the model of departure provided by Tinto (1987), we 
find this attitudinal trend particularly disconcerting. In Tinto’s 
model, the decision to persist with secondary education is lon- 
gitudinal. Over time, students’ interactions with the system and 
other individuals continually inform their evaluation to stay or 
leave (Tinto, 1987). One might infer from this model that stu- 
dents with lower attitude ratings might be subject to increased 
risk of departure. Also with respect to attitudes, literature sug- 
gests that there is a link between attitude and behaviors (e.g., 
Ajzen, 1993; Bandura, 1997). Influencing attitudes is also a 
component of long-term behavioral change (Lawless, Brown, 8c 
Cartter, 1997; Schrader 8c Lawless, 2004). In terms of academic 
and life-skills, attitude-driven improvements in behavior could 
lead to other important improvements such as graduation rates, 
GPA, or personal satisfaction. Unfortunately, the data reported 
here are insufficient to support claims about retention or perfor- 
mance without additional research into these links and possible 
outcomes of the LYE program. 

Results also indicated that there are several differences in the 
ways males and females respond to the KAB battery. On average, 
females report higher ratings in every area addressed by the LYE 
program with the exception of Proactive Behavior. Even though 
the effects are small, they range across important areas within 
the FYE program (e.g., Knowledge of Resources). Although the 
results did not suggest an interaction between gender and stu- 
dent group over time, further analysis of gender and initial group 
membership indicates that students elect to participate in the 
FYE at different rates (Pearson Chi-Square,/) = .009). A greater 
proportion of females enrolled in the FYE program. Considering 
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the positive influence of the FYE program and apparent gender 
differences, greater and more focused recruitment efforts target- 
ing males might be examined in future research. 

Although the effects of these results do not justify strong 
inferences from the data, we do not discount the potential posi- 
tive influence of the FYE program. Further, limitations of this 
investigation and design may account for the small effect sizes. 
A major limitation is selection bias. Participants were not ran- 
domly assigned to the FYE and comparison group conditions. 
Rather, they elected to participate in the FYE program. Due to 
university policies, this will likely remain the case in the future. 
However, replication of this study in a university that could ran- 
domly assign freshman to the FYE or control condition would 
provide a great deal of insight about the effect of the FYE pro- 
gram on knowledge, attitudes, and behaviors. Statistical nor- 
mality presented a problem for some of the variables. Although 
multivariate tests do not always require normality among the 
variables (Tabachnick 6c Fidell, 1996), the results need to be 
interpreted with caution. Additionally, this study was conducted 
over a single semester, which is a relatively brief span of time. It 
is difficult to conclude that a single credit course exhibits strong 
and lasting influences over knowledge, attitudes, and behaviors. 
Additional investigations are encouraged to examine the influ- 
ence of FYE programs over greater periods of time. Collectively, 
these issues may have limited our ability to measure and evaluate 
real, meaningful differences associated with the FYE. 

In response to these limitations, we suggest several topics 
for consideration and ways to improve future research associated 
with the FYE program. For example, a significant portion of the 
FYE program was focused on academic skills. Thus, the appli- 
cation of academic skills serves as a powerful metric by which 
a program may be evaluated. Future research is encouraged to 
connect the results to performance in postsecondary education 
during the first year and beyond. Further, the overall success of 
an FYE program is measured only partially in terms of learning 
and performance. In this case, one of the most powerful metrics 
concerning the FYE program would be whether or not it con- 
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tributes to the retention of students and graduation rates of the 
participants. Future research connecting the work on FYE pro- 
grams and graduation rates is recommended. Because the FYE 
is designed to be an integral part of the collegiate experience, 
these should be examined as long-term outcomes of FYE pro- 
grams. Researchers could also expand the scope and investiga- 
tion of the FYE program at the institutional, procedural, and 
policy level. In particular, it would be valuable to know what 
policy decisions and institutional supports promote the most 
successful and comprehensive FYE programs. Continuing these 
lines of research will contribute to our understanding of college 
students’ first-year experience. 


Conclusion 

This investigation was focused on implementing a compre- 
hensive evaluation of the FYE program at a large Northeastern 
university. Due to the inherent complexity of the program, the 
evaluation was designed to measure the FYE programmatic 
objectives. These objectives pertained to academic and life skills, 
particularly those associated with knowledge. By using the KAB 
battery, it was possible to observe changes in many of the factors 
associated with the FYE course. Specifically, the FYE group rated 
their Knowledge of Resources significantly higher than members 
of a comparison group. They also maintained a somewhat con- 
sistent rating pertaining to their Attitudes Toward Interactions 
while the control group’s scores declined. However, the effects of 
these findings were small and the data from this study did not 
demonstrate robust positive effects for the FYE course. 

Although it is premature to conclude that the FYE program 
is effective as measured by the KAB battery, it is equally prema- 
ture to conclude that it is ineffective. Regardless, it is likely that 
FYE interventions and other institutional supports will con- 
tinue. Research indicates that the development of student per- 
formance and the mitigation of attrition are high- stakes issues in 
secondary education. Students who do not successfully integrate 
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into the academic environment represent a loss of resources in 
terms of student talent and revenue. FYE programs like the one 
described are one aspect of universities’ solution to the problem. 
Institutions are heavily invested in programs like the FYE in 
terms of resources, time, and effort. 

Although the differences reported here may not be strong 
enough to support actionable changes, they do suggest areas for 
future investigation. There appear to be unresolved issues associ- 
ated with males and females in terms of their participation in the 
FYE program as well as their responses to the program. According 
to the data, females generally rated their scores higher than males. 
Moreover, males tend to opt out of the FYE course more often 
than females. This is a discouraging trend. Future investigations 
should examine these issues specifically. Only then would one be 
capable of recommendations, like altering recruitment strategies 
or even participation policy (i.e., requirements). Presently, these 
data and results are only suggestive of such issues. 

It is important to remember that FYE programs are intended 
and designed to benefit first-year students in a wide variety of 
ways. Ultimately, these programs are developed to prepare stu- 
dents to engage in high-level academics and to be successful in 
college (Carter et al., 2006; Ellis, 2003; VanderStoep & Pintrich, 
2008; White et al., 1995; Wratcher, 1991). This particular FYE 
program combines a variety of objectives and instructional 
approaches to facilitate student development. While these find- 
ings are not conclusive in terms of their advocacy for the FYE, 
the trends suggest a link between the FYE and important pro- 
grammatic outcomes. Although additional studies are neces- 
sary, students ultimately stand to benefit from this research and 
efforts that follow. 
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Appendix A: 

The University Experience Battery Items 


Knowledge Scale 






SD = Strongly 

Directions: Indicate your responses to the following disagree 

statements in reference to your knowledge using the SA = Strongly 
following key. Circle the appropriate response. agree 

SD 

1 

2 

3 

4 

SA 

5 

1. I know how to effectively focus my effort on my school work. 

1 

2 

3 

4 

5 

2. I know what it takes to manage my time effectively. 

1 

2 

3 

4 

5 

3. I know how to stay motivated in school. 

1 

2 

3 

4 

5 

4. When making a decision, I know how to consider ethical 

1 

2 

3 

4 

5 

factors and implications. 

5. I know how to avoid rash, spontaneous decision-making. 

1 

2 

3 

4 

5 

6. I am aware of the University counseling and support services. 

1 

2 

3 

4 

5 

7. I know how to effectively cope with academic stresses and 

1 

2 

3 

4 

5 

setbacks. 

8. I know how to resolve conflicts responsibly. 

1 

2 

3 

4 

5 

9. I know how to develop long term academic goals. 

1 

2 

3 

4 

5 

10. I am aware of my own personal limitations. 

1 

2 

3 

4 

5 

11. At the University, I know who to go to when medical issues 

1 

2 

3 

4 

5 

arise. 

12. I know how to use the library’s electronic resources to help 

1 

2 

3 

4 

5 

with my courses. 

13. When deciding which courses to take, I know who to talk to 

1 

2 

3 

4 

5 

for help. 

14. I know how to act responsibly when confronted with issues of 

1 

2 

3 

4 

5 

a sexual nature. 

IS. I know of the different signs of illicit drug use in my peers. 

1 

2 

3 

4 

5 

16. I know how to take good notes in class. 

1 

2 

3 

4 

5 

17. I know how to use computers for my courses 

1 

2 

3 

4 

5 

18. I know the sequence of courses I have to complete in order to 

1 

2 

3 

4 

5 

graduate on time. 

19. I know the value of a healthy diet. 

1 

2 

3 

4 

5 

20. I know it is important to exercise weekly. 

1 

2 

3 

4 

5 

21. I know how to solve problems. 

1 

2 

3 

4 

5 

22. I know how to work in groups. 

1 

2 

3 

4 

5 
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Attitude Scale 






SD = Strongly 

Directions: Indicate your responses to the following disagree 

statements in reference to your attitude using thefol- SA = Strongly 
lowing key. Circle the appropriate response. agree 

SD 

1 

2 

3 

4 

SA 

5 

1. Focusing my effort on my schoolwork is important to me. 

1 

2 

3 

4 

5 

2. I believe effective time management is a useful tool. 

1 

2 

3 

4 

5 

3 . I believe that my motivation for school is important. 

1 

2 

3 

4 

5 

4. When making a decision, I should consider ethical factors 

1 

2 

3 

4 

5 

and implications. 

5. I should avoid rash, spontaneous decision-making. 

1 

2 

3 

4 

5 

6. I believe it is important to use the University’s counseling and 

1 

2 

3 

4 

5 

support services. 

7. It is important to effectively cope with academic stresses and 

1 

2 

3 

4 

5 

setbacks. 

8. I should always resolve conflicts responsibly. 

1 

2 

3 

4 

5 

9. I should work towards my long term academic goals. 

1 

2 

3 

4 

5 

10. It is important to understand my personal limitations. 

1 

2 

3 

4 

5 

11. It is important to meet with the appropriate University 

1 

2 

3 

4 

5 

personnel when medical issues arise. 

12. It is important to use the library’s electronic resources to help 

1 

2 

3 

4 

5 

with my coursework. 

13. I should talk to someone to help decide which courses to take. 

1 

2 

3 

4 

5 

14. It is important to act responsibly when confronted with issues 

1 

2 

3 

4 

5 

of a sexual nature. 

15. Being able to recognize the signs of illicit drug use in my 

1 

2 

3 

4 

5 

peers is important. 

16. Good note taking skills are important. 

1 

2 

3 

4 

5 

17. Learning how to use computers for my courses is important. 

1 

2 

3 

4 

5 

18. It is important to complete courses when recommended time 

1 

2 

3 

4 

5 

in order to graduate on time. 

19.1 believe it is important to maintain a healthy diet. 

1 

2 

3 

4 

5 

20. I should exercise weekly. 

1 

2 

3 

4 

5 

21. I believe it is important to solve problems. 

1 

2 

3 

4 

5 

22. I believe it is important to work in groups. 

1 

2 

3 

4 

5 
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Behavior Scale 

NA = Not Applicable 
N = Never 


Directions: Mark the frequency that you per- R = Rarely 

form each of the behaviors listed below by circling S = Seldom 


the appropriate responses using the following O = Occasionally NA N R 

key. F= Frequently 0 12 

S 

3 

O 

4 

F 

5 

1 . 1 effectively focus my effort upon my schoolwork. 

0 1 2 

3 

4 

5 

2. 1 manage my time effectively. 

0 1 2 

3 

4 

5 

3 . 1 stay motivated in school. 

0 1 2 

3 

4 

5 

4. When making a decision, I consider important ethical factors and 

0 1 2 

3 

4 

5 

implications. 

5. 1 pause and think before I make a decision. 

0 1 2 

3 

4 

5 

6. 1 effectively cope with academic stresses and setbacks. 

0 1 2 

3 

4 

5 

7. 1 resolve conflicts responsibly. 

0 1 2 

3 

4 

5 

8. 1 strive towards my long term academic goals. 

0 1 2 

3 

4 

5 

9. 1 work within my personal limitations without taking on too much. 

0 1 2 

3 

4 

5 

10. When medical issues arise, I go (or will go) to the appropriate 

0 1 2 

3 

4 

5 

University personnel. 

1 1 . When confronted with issues of a sexual nature, I act responsibly. 

0 1 2 

3 

4 

5 

12. 1 can recognize the signs of illicit drug use. 

0 1 2 

3 

4 

5 

13. 1 take good notes in class. 

0 1 2 

3 

4 

5 

14. 1 use computers to work on my course assignments. 

0 1 2 

3 

4 

5 

IS. I maintain a healthy diet. 

0 1 2 

3 

4 

5 

16.1 exercise weekly. 

0 1 2 

3 

4 

5 

17. 1 solve problems. 

0 1 2 

3 

4 

5 

18.1 work in groups. 

0 1 2 

3 

4 

5 
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Appendix B 

Intercorrelation Matrix for the Dependent Variables 
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